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ABSTRACT 

This research study sought to determine the impact of an 
agricultural leadership program on rural community development beyond self- 
report survey data typically collected for program evaluation. Two hundred- 
ninety graduates of a program that taught leadership skills to impact policy 
at local, state, and national levels to adults aged 25-45 who were involved 
in agriculture or agribusiness were evaluated using both a quantitative mail 
survey and qualitative methods ( face-to-face interviews). The purpose of the 
study was to learn: (1) whether the program produced community development 

leaders; (2) whether participants took an active role in improving their 
communities after completing the program; and (3) whether there was a 
difference in findings based on the type of data collected. Results suggested 
that the program was an awareness program only and was marginally successful 
in integrating rural community development into the program. It was concluded 
that participants were not acting as change agents, and the program was not 
developing leaders to meet community needs. The participants were taking 
minimal leadership roles in improving their communities. In spite of 
quantitative findings of change in knowledge, skills, and behavior, 
qualitative findings did not reveal important changes, bringing into question 
the data collection methods when in-depth interviews trump survey results. 
(Includes 26 reference.) (MO) 
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Abstract 

Scope and Method of Study: The purpose of the study was to determine the impact of an 
agricultural leadership program on rural coimnunity development beyond self-report survey data 
typically collected for program evaluation. Participants in the study were graduates of the 
program from 1982 to 2002 (A=290). Quantitative and qualitative research methods were used. 
Each participant was asked to complete a then-post survey that addressed areas of knowledge 
related to rural community development, if participants were acting as change agents, knowledge 
of community needs, and leadership role in community improvement. Paired samples t-test and 
Chi-Squared tests were used to describe the data. Additionally, extreme case sampling was used 
to identify eight participants for face-to-face interviews. 

Findings and Conclusions: In spite of quantitative findings of change in knowledge, 
skills, and behavior, qualitative findings did not reveal important changes in skills or behavior 
related to leadership after completing the program. It was evident through the interviews that 
participants had not made an impact on community development. The program was found to be 
an awareness program only and was marginally successful in integrating rural community 
development process into the program. It was concluded that participants were not acting as 
change agents, and the program was not developing leaders to meet community needs. The 
participants were taking a minimal leadership role in improving their communities, bringing into 
question the data collection methods when in-depth interviews trump survey results. 

Introduction 

Agricultural leadership programs have a 70-year history in the United States. There is a 
need for leadership programs that teach citizens how to cope with the barrage of change in the 
rural environment. In particular, citizens must be educated and prepared with essential 
knowledge, skills, and abilities in order to assume leadership positions that concentrate on the 
concerns of rural America. The current array of agricultural leadership programs demonstrates a 
significant societal investment towards the important goal of fostering community participation 
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by rural citizens (Rossing & Heasley, 1987). Effective rural community development (RCD) is 
dependent on local leaders’ knowledge, skills, and willingness to assume key roles in the 
development process (Mulkey, 1989). 

Realizing the need to train more leaders to improve the quality of life for rural citizens, a 
major land-grant university in the southwest founded an agricultural leadership program in 1982. 
The goal of the program was to teach adults (ages 25-45) involved in agriculture or agribusiness 
leadership skills to impact policy at local, state, and national levels. Ten classes of approximately 
30 participants each had been completed at the time of the study. The program objectives 
included 1) increasing participants’ awareness of the agricultural industry, 2) expanding 
participants’ understanding of U.S. economic, political, cultural, and social systems, 3) 
increasing participants’ ability to analyze and react to complex problems affecting rural 
communities, 5) increasing participants’ leadership involvement and activities at the local, state, 
or national level, and 6) helping participants increase and use their skills to solve community- 
based problems. 

The program for the most recent class, held between August 2000 and March 2001, 
consisted of 13 seminars, a seven-day trip to Washington, D.C., and a two-week trip to New 
Zealand in March 2001. The weekend seminars (Friday afternoon to Sunday evening) focused on 
personal development issues, tours of agricultural research facilities, tours of specialty 
agricultural enterprises, tours of the state capital and discussions with state leaders, visits with 
agricultural association leaders and media personalities, visits to farm shows, and the future of 
rural America, including economic and demographic trends in the state. 

A review of the literature found that most evaluation studies of agricultural leadership 
programs were limited to documenting claims via self-report survey methods (Bolton, 1991; 
Howell, Weir, & Cook, 1979; Lee-Cooper, 1994; Olson, 1992; Whent & Leising, 1992). Few 
evaluation studies triangulated the data with follow-up procedures involving multiple methods 
(Rohs & Langone, 1993). Therefore, this study adds to the literature by documenting the impact 
of one adult leadership program on rural community development (RCD) using participant self- 
report data (mail survey) and face-to-face interviews as measures for understanding the 
program’s outcomes. 



Purpose of the Study 

Given the importance of effective leadership to rural community development processes 
and the challenges associated with survey data, the study asked the following questions: 

1. Did the agricultural leadership program contribute to developing leaders for 
community development? 

2. Did participants take an active role in improving their communities after completing 
the program? 

3. Was there a difference in the findings based on the type of data collected (survey vs. 
interview) in determining program effectiveness? 
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Methods for Data Collection and Analysis 



The population for the survey were all graduates of the program from 1982 to 2001 
(N-290). A census was used for the survey based on the database kept by the director. Three 
individuals were excluded from the study, due to death (n=l) and wrong addresses (n-2). 

Three data collection techniques were used for the research; 1) a then-post survey, 2) 
open-ended questions on the survey, and 3) face-to-face interviews with eight participants. Of 
the 125 participants who returned the survey (43% response rate), eight supplied extreme cases 
regarding the positive impact that the program had made on them in regard to integrating RCD 
processes into the program. Based on the survey responses, the individuals exemplified model 
change agents within their communities. Therefore, the sample for the face-to-face interviews 
was purposefully selected from subjects who completed the survey using a process known as 
extreme case sampling. Extreme case sampling involves people with unusual characteristics. In 
this case, the eight individuals were chosen based on their above average self-reported 
understanding of and commitment to RCD. 

Survey Methods 

An original survey was developed for the study based on Pigg’s (2001) work. The 
instrument was a then-post design with Likert-type scales. Respondents were asked to read each 
question, reflect on their knowledge or behavior before entering the program (then), and rate 
themselves accordingly using a Likert-type scale. A second column adjacent to the first 
contained an exact copy of the scale and asked the respondent to reflect on their knowledge or 
behavior after completing the program (post) and rate themselves a second time. The ratings 
included strongly agree, agree, disagree, and strongly disagree and were scored 1-4, respectively. 
Not sure/not applicable was coded 0 for the analysis. The two scores were compared using a t- 
test to determine differences in perception from before and after the program at a single point in 
time. The Cronbach coefficient alpha for internal consistency for all survey questions was 
calculated at 0.96. 

The then-post design was chosen to control for several challenges to validity and 
reliability, including overestimation of changes in knowledge and response-shift bias among 
participants. When pretest-posttest information is collected, actual changes in knowledge and 
behaviors may be altered if the participants overestimate their knowledge and skills on the 
pretest. Similarly, pretest overestimation is likely if participants lack a clear understanding of the 
attitude, behavior, or skill the program is attempting to affect (Pratt, McGuigan, & Katsev, 

2000 ). 



Changes in participants’ frame of reference due to the program is called response-shift 
bias (Pratt et al., 2000; Rohs, 1999). To avoid this source of error for self-report surveys, a then- 
post method was used to collect retrospective data at the conclusion of the program as 
participants rated themselves within a single frame of reference and at a single point in time. 

Although the then-posttest controls for response-shift bias and overestimation, other 
challenges to validity and reliability arise such as memory-related problems, social desirability 
responding, and effort justification (Howard, Millham, Slaten, & O’Donnell, 1981; Pratt et al.. 



2000; & Sprangers, 1987). Evaluators using retrospective tests must consider memory-related 
problems that influence the recall process. Clarifying a defined period, such as “since you began 
this program,” may facilitate recall (Pratt et al., 2000). When using retrospective tests, instead of 
representing the accurate treatment, they represent impression management as a possibility 
(Sprangers, 1987). Effort justification occurs when subjects do not experience any benefit of the 
training, and in an attempt to justify the effort spent, adjust their initial pre-treatment ratings in a 
downward direction or their post-treatment in an upward direction (Sprangers, 1987). Control for 
memory-related problems, social desirability, and effort justification was attempted by using 
objective measures (Pratt et al., 2000; & Sprangers, 1987). Interviews were also used to probe 
participants on exact behavior changes to triangulate results. 

A panel of experts consisting of four faculty members with expertise in leadership 
education or RCD processes confirmed content, construct, and face validity of the survey. A 
pilot test was conducted with 30 randomly selected participants fi'om the population. Seventeen 
people returned the pilot survey. The pilot surveys were analyzed and minor revisions were 
made. Because only minor revisions were required, the pilot data (n=17) were pooled with the 
final survey data («=108) for a final response rate of 43% (n=125). The Dillman (2000) four- 
phase mailing approach was used for both the pilot survey and the final survey. 

The double-dipping method was used to determine differences between the respondents 
and non-respondents (Linder, Miuphy, & Briers, 2001). Along with an early to late respondent 
comparison, a random sample of 10% («=20) of the non-respondents was administered portions 
of the survey via telephone. The two groups were compared on gender, employment status, level 
of educational attainment, and marital status with a Pearson Chi-Square. There were significant 
differences between non-respondents and respondents in gender, employment status, and marital 
status. There were no significant differences between the early to late respondents on any 
variable. Thus, results of the study can only be generalized to the survey respondents. 

Survey data were analyzed using SPSS® v. 8.0. An alpha level of .05 was set a priori to 
determine statistical differences among variables. The statistical tests used were descriptive, t- 
tests, and Cohen’s d effect size. Likert-t 3 q)e data is ordinal in nature; thus, it is acceptable and 
practical to treat it as interval data and subject it to statistical analysis as long as care is taken in 
the interpretation of the results (Kerlinger, 1986). Inferential statistics were used as a guide to 
understanding the relationships between variables. The effect size measures the magnitude of the 
treatment effect (Cohen, 1988). Measures of strength of association and effect size specify the 
practical significance of the research. 



Qualitative Methods 

Eight people were selected to be interviewed based on their survey responses for extreme 
cases, which demonstrated an in-depth knowledge of RCD processes. The participants were 
telephoned and asked to participate in an interview. The researcher drove to their places of 
business and conducted the interviews in their respective offices. The interviews followed a 
semi-structured outline. Probing questions allowed the researcher to explore emerging themes 
and to confirm hypotheses (Merriam, 1998). 

To establish validity for the interviews, each interview was recorded and transcribed. The 
transcriptions were sent to the interviewees to validate their statements (Merriam, 1998). The 
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qualitative analysis software program ATLAS. ti® was used to organize the data fi'om the open- 
ended survey questions and the interviews. Both data sets were analyzed and reported following 
Creswell’s (1998) procedures: 

1 . Organization of data. The interviews were recorded and transcribed, cleaned by a 
research assistant who listened to the interview and read the transcript to check for 
accuracy. The text was then loaded into the qualitative data program ATLAS.ti®. 

2. Categorization of data. The data were clustered into meaningful groups (coded) using 
ATLAS.ti® as an organizational tool. 

3. Interpretation of the data. Statements that fell into like codes were examined for specific 
meanings in relationship to the purpose of the study. 

4. Identification of patterns. The data and their interpretations were examined for themes 
and patterns that characterized the program and allowed the researchers to draw 
conclusions. 

5. Synthesis. An overall representation of participants’ responses was created where 
conclusions and recommendations were drawn based on the data presented. 

Findings and Conclusions 

Respondents ’ Profile 

Survey respondents were married (90%), well-educated, middle class working adults who 
were civically engaged. One-hundred and thirteen men (90%) and 12 women (10%) responded to 
the survey. Their mean age was 43 years. The majority (54%) graduated college and 32% had 
earned graduate credit. Forty-seven percent earned $30-$50,000 annually and 100% voted in the 
last presidential election. Sixty percent volunteered 5-10 hours per month in social service 
activities and 69% were involved in 5-10 hours of economic development activities per month. 
They lived in their communities for an average of 24 years and the average community size was 
30,000 people. 

Did the agricultural leadership program contribute to developing leaders 
for rural community development? 

Before community leaders can implement desired change, they must have a feel for 
existing attitudes and perceptions with respect to those factors that impact economic 
development objectives and outcomes (Williams, 1989). Effective community leaders could also 
promote community development by determining what leadership styles are needed for change 
based on their own, and their followers’, skills and education (Robinson, 1994). Community 
leaders should be able to identify problems, assess community organizational structures, develop 
the necessary capacity, and design a plan for action to address problems (Mulkey, 1989). 

The survey findings indicated that respondents believed the program developed them as 
leaders to meet their community’s needs. A paired samples t-test resulted in significant 
differences for each variable fi'om the then-post survey. The effect size, Cohen’s d, was 1.79, 
indicating a large effect size for this construct (Cohen, 1988). 
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Findings from the eight purposefully selected interviewees were synthesized and 
conclusions are presented to triangulate the survey findings with additional data sources. 
Interviewees are represented by numbers presented in brackets to protect their identity. 

Successful community development efforts are largely dependent on locally generated 
knowledge of how the community works. The development process includes needs assessment, 
community analyses, consensus building, and goal setting. Where these activities exist, 
communities are more likely to be actively engaged in the process of discovering and 
understanding their needs (Mulkey, 1989). 

All eight interviewees were asked directly if they could identify their communities’ needs 
[1, 29, 90, 134, 168, 208, 272, and 290]. Three participants stated that the program showed them 
who they needed to contact so those needs could be identified [1, 168, 290]. These participants 
believed that the program focused on how to find information, not how to use the information to 
implement change. “The one thing I learned in [the program] is I don’t have to have those skills 
(RCD); I just need to know where to go to get them” [168]. “The groundwork was laid so we did 
learn whom we needed to talk to so we can find out those needs” [290]. 

One of the most important components of community leadership is the ability to generate 
collective action at the community level (Heekathom, 1993). A central concept in the RCD 
literature emphasizes the importance of local participation as a means of strengthening the 
community (Martin & Wilkinson, 1985). One participant believed the program taught him to 
seek political power at the national level rather than work for development at the local level. “It 
probably helped me a lot more at the state and national level than on a community level. 
Basically, a lot of the things have a reflection on me and what’s going to pay off on me is not as 
much at the local level as a state or regional level. The program identified more in what to do in 
the political process, more of how to sequester groups to help you with some of your problems 
and how to look at some groups that have similar causes to try to get those groups together 
because more numbers mean more votes for elections and people get their way” [90]. Participant 
90’ s preception that the program helped more at the state and national level is inconsistent with 
the literature concerning effective RCD. 

Three participants [29, 168, 272] believed they had a good understanding of their 
communities’ needs before entering the program. These participants did not believe the program 
changed their knowledge of community needs. One interviewee did not think the program gave 
him the skills to identify needs in his community [134]. 

The participants were asked what the program could do to teach them to learn how to 
identify their community’s needs. Two participants believed that the seminars should be changed 
to develop skills rather than focus on awareness [134, 208]. “Bring the whole aspect of 
community development into the program. Change the focus of the program to teach participants 
how to identify what the needs of their communities are; talk more about the different aspects of 
local government and organizations” [208]. 

Before community leaders can implement change, they must have a feel for existing 
attitudes and perceptions with respect to those factors which impact development (Williams, 
1989). After reviewing the literature, it was concluded that if the program were developing 
leaders to meet community needs, then participants would know how to identify those needs as 
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well as design and implement action plans for community development. This was not the case 
among the eight interviewees. Therefore, the program did not contribute to developing leaders 
for rural community development as participants were not equipped with the knowledge or skills 
to identify their community’s needs. The program did, however, increase awareness that 
communities have needs. 

Did participants take an active role in improving their communities 
after completing the program? 

The importance of participation as a means of strengthening local conummities cannot be 
overstated as commxmity leaders provide the basis for improving the quality of life in rural 
America (Martin & Wilkinson, 1985). People must not only get involved, they must also recruit 
people from racial, ethnic, and socioeconomic backgrounds who represent the community 
(Beaulieu & Smith, 2000). 

The survey findings indicated that respondents believed they were taking an active role to 
improve their conummities. A paired samples t-test resulted in significant differences for all but 
one variable from the survey. The effect size, Cohen’s d, was .67, indicating a large effect size 
for this construct (Cohen, 1988). The insignificant variable was, “I am very active in making 
efforts to improve the well being of the disadvantaged in my community” (p=0.15). 

The survey findings also indicated that respondents believed they were listening to people 
with different socioeconomic status within their communities. A paired samples t-test resulted in 
significant differences for all variables from the then-post survey. The effect size, Cohen’s d, 
was .49, indicating a medium effect size for this construct (Cohen, 1988). 

All interviewees were asked specifically what they had done to improve their community 
since completing the program [1, 29, 90, 134, 168, 208, 272, and 290]. Five interviewees 
reported they had not been active in their conummity as leaders [29, 90, 208, 208, and 290]. “I 
probably have not done as much as I potentially could in developing this community” [90]. “I am 
not taking on as much as I probably should have” [29]. “I am not very active as far as a 
community leader in community organizations. I hope that I have become more active in my 
community in more of a support role. I don’t feel like I came home and became a driving force to 
develop local communities [after the program]” [290]. One interviewee believed he was more 
involved in leadership roles before the program than after [208]. He believed his opinions were 
drastically different from other people and that the only leadership role he could assume was to 
lead by example and change his agricultural operation to be more sustainable [208]. 

Involvement at the local level was problematic for one participant [90]. He did not 
believe that graduates from the program could effectively be involved in community 
organizations because the graduates are more developed and better-quality leaders than ones in 
local community organizations who have not participated in the program. “Getting involved in 
the local organizations is probably a loser. The people who graduate from the program are so far 
ahead and the local agenda is so slow.. . . The people who graduate from [the program] are 
motivated by what helps them and their families” [90]. Another interviewee [168] believed the 
program stressed involvement at the state level instead of the community level. 
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Two participants have taken on leadership roles in regional organizations [1, 134], and 
one participant [168] has started working on developing local projects to benefit the commvinity. 
“I have taken on new leadership roles in regional organizations, I wouldn’t have [done this ] had 
I not gone through the program, but I could make more of an impact on rural development if I 
had more skills in managing change, strategic planning, and needs assessment” [134]. 

Networking opportunities was the most important aspect of the program for all the 
interviewees [1, 29, 90, 134, 168, 208, 272, and 290]. The exposure to different people and 
organizations put participants in contact with people who could assist them in RCD efforts. 
However, they did not believe they were currently using their networks to the fullest extent 
possible at the time of the interview [1, 29, 90, 134, 168, 208, 272, and 290]. 

Qualitative data fi'om the open-ended questions on the survey were used to triangulate 
findings. Sixty- four respondents (51%) answered the question on the survey: “What was most 
beneficial to your community development efforts?” Eighteen of the sixty-four respondents 
(28%) believed that networking was the most beneficial thing they learned in the program. The 
networks offered exposure to other people and were valuable for direction and support [15, 179]. 

Based on the interview responses, most participants were not making changes in their 
communities, nor had they used their networks for community improvement. Participants 
reported that information gained in the program was not effectively used because they did not 
have the necessary skills to promote change. 

The RCD process includes problem and needs identification, assessment of community 
organizational structures, developing capacity, and implementing programs to address issues 
(Mulkey, 1989). Community leaders should have adequate knowledge and skills to carry out 
these functions. One of the most important components of community leaders is the ability to 
mobilize resources at the community level (Heekathom, 1993). The RCD literature emphasizes 
the importance of local participation as a means of strengthening the local community (Martin & 
Wilkinson, 1985). When asked directly about involvement in RCD activities, participants 
reported not being active in any phase of community development. Therefore, it was concluded 
that participants had not taken an active role in improving their commvinities after completing in 
the program. 



fVas there a difference in the findings based on the type of data collected 
(survey vs. interview) in determining program effectiveness? 

All variables for the then-posttest survey were statistically significant at the .05 level, 
indicating that participants perceived they had gained knowledge and skills fi'om the agricultural 
leadership program. However, when the eight purposefully selected participants were asked 
about their imderstanding of rural community development processes, it was foimd that they 
were not participating actively in community development activities, thus, they were not acting 
as change agents in their communities. 

When comparing the findings fi'om the survey data vs. the interview data, it can be 
concluded that the survey respondents overestimated their knowledge and skills regarding RCD 
processes on the survey (Pratt et al., 2000) due to social desirability (Howard, et al., 1981) and 
effort justification (Sprangers, 1987). 



ERIC 



8 



10 



The study should alert other researchers’ attention to the fact that self-report survey 
methods of evaluation may be inadequate for determining program impacts. Participants could 
not authenticate actual changes in behavior made after participating in the program. Survey- 
based studies may actually be documenting participants need for effort justification rather than 
tangible program impacts. 

Recommendations, Discussion and Implications 

The agricultural leadership program did create awareness among participants regarding 
the importance of RCD as stated in the objectives; however, it failed to move participants into 
action by producing community leaders. The qualitative data suggested that awareness was 
inadequate for participants to lead community development efforts as participants lacked both 
knowledge and skills for effecting change. Program designers should move beyond providing an 
awareness only program and provide opportunities to increase participants’ skills in RCD 
processes by integrating more seminars and workshops into the program that focus on the 
mechanics of RCD. These experiences should also focus on new development opportunities 
where participants can engage in discussions with successful community leaders. 

Townsend (2002) reported that one-shot programs develop awareness but were not 
effective in changing behavior. When an extended and sustained leadership class was provided, 
attitudes and leadership behaviors changed after the class. The agricultural leadership program 
used in this case study provides the long-term contact needed to change behavior; thus, the 
potential for incorporating knowledge and skill development exists but is currently under 
utilized. Program designers should integrate a leadership project or practicum into the program. 
Asking participants to create and implement a plan for community development within their 
home towns would serve to develop leadership skills, needs assessment skills, change agent 
skills, and increase participant impact on community development, at least in the short term. By 
experiencing success in a community development project, participants may also become more 
motivated to repeat the experience and become truly effective leaders rather than bystanders in 
their communities. 

Other methods to determine participant impact on commimity development should be 
used to triangulate self-report survey data such as observation, interviews with participants and 
other community members, and collecting data other than participant satisfaction with the 
program. Program evaluators should also considering abandoning self-report survey research in 
favor of more credible data if funds for evaluation are limited. The financial and human 
resources used in developing the survey for this study could have been used toward randomly 
selecting more interviewees for face-to-face interviews as this study foimd that the survey data 
was invalidated by the in-depth interviews. 

Recommendations for further research include conducting a longitudinal study of the 
program to document changes in the program based on the initial findings using interviews and 
observations as primary data sources. Also, the program designer should incorporate a 
participant-centered documentation process of the participants’ impact on community 
development for internal evaluation purposes. 
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